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study indicates that the MMAI is an effective and reliable instrument for 
measuring the extent to which curricular materials meet NCTM standards. 
Contains 80 references; the MMAI for grades 5-8 and 9-12; and instructions 
for coding, calculating, and interpreting the MMAI. (LP) 



******************************★★**************************+****************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



ED 408 133 



i 



A Content Analysis Study of Portable Assisted Study Sequence 
Mathematics Curricular Materials for Migrant Students 
using the National Council of Teachers of Mathematics Standards 
Karen I. Conger 
Walden University 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

ter This document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



“PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



j<arer\ Z 

Conger 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC).” 



Author Note 

Karen I. Conger received the Ph.D. in Education (1996) from Walden University, Minneapolis, 
MN. She serves as adjunct mathematics faculty for University of Phoenix in central California, and 
secondary mathematics teacher for Kern High School District in Bakersfield, CA. She is also a training 
consultant for QSR Nud*Ist qualitative data analysis software, and writer of mathematics curricular 
materials. 




O 

MKIC 



BEST COPY AVAILABLE 



2 



2 



Abstract 

Mathematics curricular materials are often written to reflect reform standards to serve the future 
needs of the student population. This paper discusses the findings from the researcher’s doctoral 
dissertation that involved content analysis and comparisons of Portable Assisted Study Sequence 
(P.A.S.S.) mathematics curricular materials designed for the migrant student population. This study was 
undertaken because no research had analyzed mathematics materials written for the migrant student 
population with respect to National Council of Teachers of Mathematics Curriculum and Evaluation 
Standards for School Mathematics (1989). Furthermore, an evaluative instrument designed to measure 
the extent to which reform ideas in the Standards are represented in the materials did not exist. These are 
needed because it is becoming increasingly important to evaluate curricular materials with respect to the 
Standards if we intend to radically change our mathematics curriculum. 

The migrant student represents one segment of the student population with deficiencies in 
mathematics training at the K-12 level. The P.A.S.S. Program serves migrant students in 165 schools in 
California as well as schools across the nation, and is legally bound to provide materials meeting reform 
standards. The 1995 curricular materials were compared through content analysis to the 1989 materials 
they replaced with respect to the Standards. This required the design and use of an evaluative instrument. 
Mathematics Materials Analysis Instrument (MMAI), to determine whether it is possible to measure the 
extent to which the materials are meeting reform standards. 

Nonparametric and qualitative analysis methods, including the use of Nud*Ist qualitative data 
analysis software, were used in this study. The researcher found the 1995 P.A.S.S. curricular materials 
measurably improve upon the 1989 materials with respect to the Standards, and the evaluative instrument 
MMAI effectively and reliably measures the extent to which curricular materials meet the Standards. 

The researcher concluded that the studied 1995 P. A.S.S. curricular materials are reflecting reform 
standards, and that an instrument now exists to quantitatively measure the extent to which reform ideas in 
the Standards are represented in the materials. This instrument can be improved upon, and the researcher 
acknowledges this study represents a pioneering effort in the mathematics reform movement. 
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Introduction 

Mathematics curricular materials are often written to reflect reform standards to serve the future 
needs of the student population. Migrant students represent one segment of the student population with 
deficiencies in mathematics training at the K-12 level. The Portable Assisted Study Sequence (P.A.S.S.) 
Program serves migrant students in 165 schools in California as well as schools across the nation. The 
P.A.S.S. Program designs curricular materials for this population and is legally bound to provide 
materials meeting the California mathematics framework. This framework reflects ideas and concepts 
envisioned in the National Council of Teachers of Mathematics Curriculum and Evaluation Standards for 
School Mathematics (1989). 

This paper discusses the findings from the researcher’s doctoral dissertation that involved content 
analysis and comparisons of Portable Assisted Study Sequence (P.A.S.S.) mathematics curricular 
materials designed for the migrant student population. This study was undertaken because no research 
had analyzed mathematics curricular materials written for the migrant student population with respect to 
National Council of Teachers of Mathematics Curriculum and Evaluation Standards for School 
Mathematics (1989). Furthermore, an evaluative instrument designed to measure the extent to which 
reform ideas in the Standards are represented in the curricular materials did not exist. These are needed 
because it is becoming increasingly important to evaluate curricular materials with respect to the 
Standards if we intend to radically change our mathematics curriculum. 

The 1995 P.A.S.S. curricular materials were compared through content analysis to the 1989 
P.A.S.S. materials they replaced with respect to the Standards . This required the design and use of an 
evaluative instrument. Mathematics Materials Analysis Instrument (MMAI) (Appendix A), to determine 
whether it is possible to measure the extent to which the curricular materials are meeting the Standards . 

Background and Current Research 

Mathematics Reform Standards. The reform movement in mathematics is not new; it changes from 
decade to decade depending on societal needs and influences. These needs and influences are economic. 
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political, social, environmental and so forth. Our society has required mechanistic types of output and 
thought processes for several decades. Futurists believe productivity in the 21st century will depend upon 
a technologically literate, creative workforce able to work in multicultural teams, as well as independently, 
to solve global problems. These workers will need to consider the full ramifications of their decisions 
across all disciplines rather than the narrow focus of their own environment. The mathematics 
curriculum must provide for these expectations by helping students learn mathematics as a process of 
experimenting, abstracting, generalizing, and specializing rather than a litany of extolled facts and 
theories. 

The reform standards that are currently shaping U.S. mathematics curriculum are the Curriculum 
and Evaluation Standards for School Mathematics (National Council of Teachers of Mathematics, 1989). 
The Standards as the document is called, are designed to move mathematics curriculum forward to meet 
the needs of students for the future. A majority of states are now involved in the development, revision, 
and implementation of state frameworks in mathematics, and many mathematics frameworks agree with 
the recommendations of the Standards 

The Migrant Student and Immigration 

The migrant student is a special concern in education. The migrant student is the child of 
migratory workers who are often fishermen or agricultural workers, and they move from one state or 
district to another for the purpose of finding temporary or seasonal employment (Cahape, 1993; California 
Department of Education, Handbook, 1992; U. S. Department of Education, 1985). Velazquez (1994) 
discusses the movement of migrant workers along three identifiable streams: Eastern, Mid-Continent, and 
Western. The Western stream is the largest, "extending from California and Arizona to Oregon and 
Washington" (p. 32). 

Many immigrants are migratory workers, and many of the immigrant migratory workers are 
illegal. There is some disagreement on statistics relating to these workers. Schuck (1995) believes that 
"the vast majority of aliens [foreigners] who enter illegally are more or less seasonal migrants" (p. 90). 
Trotter (1992) estimates that 95% of illegal immigrants are farm workers, and 90% of these are Latino. 
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SchJosser (1995) maintains that 30% to 60% of the migrant workers in California are illegal immigrants. 
Velazquez (1994) claims undocumented workers comprise only 15% of all migrants. Doyle (1990) cites 
studies that support this smaller number, and claims that many of the labeled migrant students maintain 
stable residences. 

California is presently struggling with the issue of illegal immigration due to the great influx of 
two to three million illegal residents (Cox News Service, 1997; Schuck, 1995; O'Halloran, 1994). 
Proposition 187, approved by California voters in November 1994, is an anti-illegal immigrant initiative 
that has caused much controversy and alarm throughout the United States. It seeks to eliminate 
educational, medical, and welfare funding for illegal residents, with many Mexican-Americans supporting 
the proposition (Schuck, 1995). Support for the proposition is fueled by the illegal immigrant's fierce 
allegiance to Mexican nationalism and defiance of Americanization (defined by Aldama, 1995, as 
assimilation, acculturation, and citizenship.) The California "voters responded angrily to the vivid 
television images of Mexican officials denouncing the measure and to the marchers in Los Angeles 
waving Mexican flags and protesting its limits on welfare benefits" (Schuck, 1995, p. 90). Proposition 
187 is being challenged in court by opponents who claim it violates federal and state guarantees of equal 
protection, of state and federal privacy rights, and of international law. The federal courts have 
temporarily blocked the amendment as many U.S. citizens favor continuance of public benefits for illegal 
immigrants. California laws specifically give its citizens "a right to a basic education and the Legislature 
has a constitutional duty to provide one" (O’Halloran, 1994, p. 370). 

Migrant workers and their families are in the midst of this battle. Schlosser (1995) cites the 
contribution of illegal immigration to the political and economic well being of California. Agriculture is 
California's largest industry and it now produces "more than half the fruits, nuts, and vegetables 
consumed in the United States" (Schlosser, 1995, p. 80). Schlosser maintains that 30% to 60% of the 
migrant workers in California are illegal immigrants, and that "illegal immigrants, widely reviled and 
depicted as welfare cheats, are in effect subsidizing the most important sector of the California economy" 
(p. 82). In fact, illegal immigrants are so essential to the U.S. agricultural economy that legislators often 
find ways to provide temporary guest worker programs for states that are dependent upon them. "Skillful 
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manipulati on of an increasingly vulnerable administrative system" (Scliuck, 1995, p. 92) provides that the 
legal status of aliens "who enter surreptitiously should be called 'undocumented' rather than 'illegal' 
because their legal status remains uncertain for months or years during which the aliens can usually 
obtain work permits" (p. 92). 

The stark truth is that the agricultural employment is a lifeline to migrant families. The cheap 
wages in the U.S. are up to 10 times the wages earned by Mexican peasants in their native villages. It is 
at a cost to Mexico and the United States. Mexico loses its surplus workers and the United States 
increasingly pays higher costs as migrants marry and raise children within the U.S. The Immigration and 
Naturalization Service "has traditionally rounded up and deported illegal immigrants in California 
immediately after the harvest" (Schlosser, 1995, p. 99). The workers who are overlooked often become 
American citizens and eventually find less physically demanding and more financially rewarding kinds of 
work in factories and other skilled trades. "As a result, the whole system now depends on a steady supply 
of illegal immigrants to keep farm wages low and to replace migrants who have either retired to Mexico 
or found better jobs in California" (p. 99). 

The types of problems faced by migrant students are varied. Prewitt-Diaz (1991) lists four factors 
affecting migrant children in school: ecological, educational, psychological, and economical. Many 
families are seeking refuge from tyranny in their native countries. Others seek a better lifestyle and more 
job opportunities. Children move regularly from district to district, and experience absenteeism and 
falling behind in academic areas. Their self-image is affected as they struggle with their language and 
new relationships. Cultural differences create problems of inclusion within the classroom. Children are 
contributors in their families and "are essential in the economy of the migrant family" (Prewitt-Diaz, 

1991, p. 485). They have power and may control their parent(s) as they become the interpreters between 
the school and the home. Romo ( 1 993) lists similar problems, and adds that many secondary-age 
students have only attended grades 1-6 in Mexico. Velazquez (1994) describes their feeling of 
powerlessness combined with their respect for authority. Families have little formal education and trust 
the schools. They "feel that their questions about the appropriateness of their children's educational 
program will be construed as a challenge to the teacher's authority and prestige" (p. 33). 




7 



7 



The impact of the migrant student on our schools is reflected by the rapid changes occurring in 
our public school population. Again, statistics differ. Estimates range from half a million to as many as 6 
million migrant students enrolled in public education in the U.S. and Puerto Rico. Cahape (1993) reports 
there are over half a million migrant children enrolled in public education in the 50 states, the District of 
Columbia, and Puerto Rico. O'Halloran (1994) claims more than two million of those enrolled in public 
institutions in the last decade were immigrant youth, and 70% reside in just five states, "the majority 
having settled in California" (p. 371). Headden (1995) reports nearly three million students, mostly 
Latino, in the American educational system are designated as limited English proficient (LEP). Nearly 
45% of these students live in California, and many of these children are migrant students. Trotter 
( 1 992) argues that many migrant children are unidentified and believes "estimates of all those engaged in 
migrant labor range between 1.7 million and 6 million" (p. 16). 

The most accurate statistics may be kept by the Migrant Student Record Transfer System 
(MSRTS). MSRTS is a computerized information network used by approximately 17,000 sites in the U.S. 
that regulates and transfers data on migratory students as they move from school site to school site. 
MSRTS figures for 1990 show there are approximately 600,000 migrant children in the U.S. with the 
following concentrations: California (209,006), Texas (123,187), Florida (59,195), Washington (30,000), 
Arkansas (20,000), Oregon (20,000), New York (10,000), and the least. District of Columbia (190), and 
West Virginia (94). 

The diffi culty in compiling accurate statistics regarding migrant students is explainable. Federal 
program regulations require state Departments of Education to identify and educate migratory children. 
Authorized recruiters for the migrant student programs identify these students, but many students are not 
found. The California Handbook for Identification and Recruitment (California Department of Education, 

1 992) discusses the difficulty of finding children in rural settings who may be living temporarily in 
abandoned buildings, orchards, and cars. 

The "culture of migrancy" (Velazquez, 1994, p. 32) contributes to the difficulty. Children 
atcmrifi adult roles in the fields, and "most migrant children drop out of school when they are able to work 
in the fields and earn money" (p. 33). The Migrant Student Record Transfer System (MSRTS) reports 
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that the drop-out rate for migrant students is between 35% and 60%, and that most have dropped out by 
10th grade. Overby (1993) reports dropout rates for migrant students of 43% as compared to Mexican- 
Americans of 35.8%. This reflects an improvement for migrant students over previous dropout rates as 
high as 90% in the 1970s (Cahape, 1993), however, and graduation rates have also increased. "Between 
1984 and 1990, the number of migrant students enrolled in 12th grade climbed from 21,493 to 30,745-a 
43% rise" (Trotter, 1992, p. 17). Trotter points out that most do not graduate, however, and that student 
enrollment had actually increased by 13% during the same period. He reports only 13.8% of migrant 
students graduate, compared to 87.8% of the general population, and 67.6% of the Latino population. 

Grade level retention rates are also a problem. Migrant students are retained at grade level at 
least 1 year twice as often as the general population, largely due to academic deficiencies that result from 
problems associated with their lifestyle. MSRTS reports "33% are one year below grade level and 17% 
are two years or more below grade level" (Cahape, 1993). 

The federal government remains dedicated to its commitment to migrant children and families as 
demonstrated by the government's blocking of Proposition 187 and other legislation. The Migrant 
Education Program was authorized in 1965 through the Elementary and Secondary Education Act. 

Federal program regulations require state Departments of Education to identify and educate migratory 
children. The California Department of Education assumes responsibility for all statutory and regulatory 
requirements of the program including subgrantees. Funding is based on a "Full-time Equivalent (FTE) 
count of each individual child for each day of residence in the State. This count is based upon the entry of 
data into the Migrant Student Record Transfer System (MSRTS) for each State for each year" (California 
Department of Education, Handbook, 1992, p. 1-2). 

California Portable Assisted Study Sequence (P.A.S.S.l Program. The California Portable 
Assisted Study Sequence (P.A.S.S.) program is based in part on a newer Federal Law, P. L. 100-297, 
which was passed in 1988, and California Assembly Bill No. 1382, which was passed in 1981. This 
program serves 165 schools in California and serves as a national center for migrant student curriculum. 
P.A.S.S. is accredited through Fresno Unified School District, and the Western Association of Schools 




and Colleges. The 1993-1994 National Report for the California Portable Assisted Study Sequence 
(P.A.S.S.) Program, which serves migrant students in California, shows it served 8,326 of the estimated 
209,006 migrant students in California in 1993-1994. Nearly all of these students (8,243 or 99%) were 
Latino. 

The program allows migrant students to carry portable work-text units from one school site to 
another. These courses are accepted as credits at participating school sites, and therefore, migrant 
students are able to transfer credits within the state and to at least twenty other states. The portable units 
can be continued at new school sites and students receive graduating credits. Therefore, the P.A.S.S. 
Program must not only provide materials that will be appropriate for the migrant student, but must also 
reflect mathematics reform standards. 

The P.A.S.S. Handbook states that its courses "have the same content as the regular high school 
courses" (p. 2). The mathematics courses are sequential. The first courses in the sequence, written in 
1989, are General Math A and General Math B. They were updated in 1995 as Integrated Math A and 
Integrated Math B. Other math courses include Consumer Math . Pre- Algebra , and Algebra A and 
Algebra B . A fifth course, Geometry is planned for late 1997. The Consumer Math course was rewritten 
in 1996-1997 as Consumer Education to integrate new reform ideas with career math into the program. 

Methodology 



Data sources 

The curric ular materials analyzed for this study were the 1989 P.A.S.S. curricula entitled General 
Math A and General Math B. and the 1995 P.A.S.S. curricula entitled Integrated Math A and Integrated 
Math B . The 1995 materials were developed to replace and update the 1989 materials in relationship to 
the goals and spirit of the National Council of Teachers of Mathematics Curriculum and Evaluation 
Standards for School Mathematics (1989). 
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Study Questions 

Two questions addressed in this study were: 

1. To what extent do the 1995 P.A.S.S. curricular materials improve upon the 1989 materials in 
reflecting reform ideas expressed in the Standards ? 

2. Can a researcher-designed evaluative instrument measure the extent to which curricular 
materials meet the Standards ? 

Instrumentation 

An evaluative instrument was designed for this study to quantify the relationship between 
mathema tics materials and the recommendations in the NCTM Standards . This instrument is entitled 
Mathematics Materials Analysis Instrument (MMAI) (Appendix A) and is divided into two grade levels: 
grades 5-8 and grades 9-12. The instrument is a modification of guides from those grade levels for the K- 
12 mathematics program found in A Guide for Reviewing School Mathematics Programs (NCTM, 1991). 
The modification process was accomplished through the joint efforts of the researcher and an expert 
validation panel consisting of three educators who are familiar with, and experienced in, the vision of the 
NCTM Standards . 

The evaluative instrument MMAI consists of eight categories (with 3 to 12 subcategories each) 
for grades 5-8 and eight categories (with 6 subcategories each) for grades 9-12. The instrument includes 
an ordinal value scale: "1-None, 2-Low, 3-Moderate, 4-High." These ordinal values reflect the user's 
perception of the extent of alignment of content in the targeted curricular materials to reform standards 
delineated in the subcategories on MMAI. 

Interrater reliability was established by measuring the extent to which similar ordinal values were 
assigned by various coders at separate locations using the evaluative instrument MMAI. This involved 
testing to see if there were significant differences in the coding assigned by two human coders (Coder 1 
and Coder 2) and Nud*Ist computer coding (Coder 3) using MMAI on the 1989 and 1995 P.A.S.S. 
materials. The means of ordinal values assigned by each coder were computed and ranked for the 1989 
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and 1995 curricula at each grade level 5-8 and 9-12. The nonparametric Kruskal-Wallis H-test was then 
performed for each curriculum at each grade level, and no significant differences were found (Tables 1-2). 



Table 1 



Kruskal-Wallis H-test to Establish Interrater Reliability Between Coders Using the 



Mathematics Materials Analysis Instrument (MMAI) on 1989 Curriculum 



MMAI Category 
Rank 


Coder 1 


Rank 


Coder 2 i 


Rank 


Coder 3 


Grades 5-8 














A 


2.08 


11.5 


2.08 


11.5 


2.17 


13.0 


B 


1.71 


4.5 


1.86 


6.5 


1.86 


6.5 


C 


2.25 


16.0 


2.25 


16.0 


2.25 


16.0 


D 


2.44 


20.0 


2.22 


14.0 


2.56 


21.0 


E 


2.40 


19.0 


2.60 


22.5 


2.60 


22.5 


F 


1.71 


4.5 


2.00 


9.0 


2.29 


18.0 


G 


1.00 


2.0 


1.00 


2.0 


1.00 


2.0 


H 


2.00 


9.0 


2.00 


9.0 


3.33 


24.0 


H = 2.004 a = 


N, =8 
.05 


R wm =86.5 


£ 

II 

00 


R«m, = 90.5 


N 3 = 8 


R«m,= 123 



MMAI Category 
Rank 


Coder 1 


Rank 


Coder 2 




Rank 


Coder 3 


Grades 9-12 
A 


1.83 


13.0 


2.17 


22.0 


2.33 


24.0 


B 


1.67 


9.0 


2.00 


18.0 


1.83 


13.0 


C 


2.00 


18.0 


2.00 


18.0 


1.50 


6.5 


D 


1.83 


13.0 


1.83 


13.0 


2.17 


22.0 


E 


1.33 


5.0 


1.17 


4.0 


1.50 


6.5 


F 


1.67 


9.0 


1.83 


13.0 


2.17 


22.0 


G 


1.00 


2.0 


1.00 


2.0 


1.00 


2.0 


H 


2.00 


18.0 


1.67 


9.0 


2.00 


18.0 




N, = 8 


R«um = 87 


QO 

II 

n 

z 


R«um 


N 3 = 8 
99 


R«um = H4 



H = 0.915 a =.05 




Table 2 
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Kruskal-Wallis H-test to Establish Interrater Reliability Between Coders Using the 
Mathematics Materials Analysis Instrument (MMAI) on 1995 Curriculum 



MMAI Category 
Rank 


Coder 1 


Rank 


Coder 2 


Rank 


Coder 3 


Grades 5-8 














A 


3.42 


17.0 


3.25 


11.5 


3.58 


22.0 


B 


3.14 


8.5 


3.57 


20.0 


3.29 


13.0 


C 


3.75 


23.0 


3.25 


11.5 


3.50 


19.0 


D 


2.78 


4.0 


2.44 


2.0 


2.56 


3.0 


E 


3.40 


16.0 


3.20 


10.0 


3.30 


14.0 


F 


3.43 


18.0 


3.14 


8.5 


3.57 


20.5 


G 


2.22 


1.0 


3.00 


6.5 


2.89 


5.0 


H 


3.33 


15.0 


3.00 


6.5 


4.00 


24.0 


H = 2.389 a = 


N, = 8 
.05 


N 2 =8 

R«»m= 102.5 


N 3 = 8 

R«um = 77 


R«um = 120.5 



MMAI Category 
Rank 


Coder 1 


Rank 


Coder 2 


Rank 


Coder 3 


Grades 9-12 














A 


3.17 


21.0 


3.00 


17.0 


3.83 


24.0 


B 


3.00 


17.0 


3.00 


17.0 


3.00 


17.0 


C 


2.33 


1.5 


2.50 


3.5 


2.83 


10.5 


D 


2.83 


10.5 


3.00 


17.0 


3.00 


17.0 


E 


3.50 


22.0 


2.50 


3.5 


3.67 


23.0 


F 


2.33 


1.5 


2.67 


6.0 


2.67 


6.0 


G 


2.83 


10.5 


2.83 


10.5 


3.00 


17.0 


H 


2.83 


10.5 


2.67 


6.0 


2.83 


10.5 




N, = 8 


N 2 =8 

R«um=94.5 


N 3 = 8 

R«um=80.5 


R«um = 125 



H = 2.589 a = .05 



Content Analysis Methodology 

This study utilized the methodology of content analysis. Triangulation (Kelle, 1995; Patton, 
1990) in this study consisted of narrative descriptions and comparisons, manual data collection and 
coding, and computer analyses using Nud*Ist qualitative data analysis software (Richards & Richards, 
1995). 
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The evaluative instrument designed for this study, Mathematics Materials Analysis Instrument 
(MMAI). (Appendix A) was used to obtain ordinal data from the 1989 and 1995 curricula. Coding was 
performed by three coders (two human coders and Nud*Ist computer coding). These data were converted 
to statistical measures of dispersion for analysis. 

Data was collected from data collection worksheets and word sort lists containing frequency 
counts for each curriculum. These word lists were manually examined for key word-in-context listings, 
classifications of words into content categories, content category counts, and retrievals based on content 
categories and co-occurrences. The 1989 and 1995 curricula were also examined by comparing titles, 
subheadings, sections headings, student directions, and teacher guidelines. Discoveries made during this 
process led to further examination of classifications of words into content categories, content category 
counts, and other interpretative analyses (Krippendorff, 1980; Weber, 1990). 

Findings and Conclusions 

Nonparametric and qualitative analysis methods were used to analyze the data in the study and 
interpret the results. 

Study Question 1 : The first study question "To what extent do the 1995 P. A.S.S. curricular 
materials improve upon the 1989 materials in reflecting reform ideas expressed in the Standards ?" was 
answered in several ways. 

Chi-square analysis showed a statistically significant difference between the frequencies of 
ordinal value coding and the 1989 and 1995 curricula in relationship to the Standards (Table 3). 
Frequency counts for each of the three coders were computed from tallies of coded values on MMAI for 
the 1989 and 1995 curricula for each grade level 5-8 and 9-12. Total frequencies and expected values 
were calculated for each coder using 4x2 contingency tables. The null hypothesis was rejected at both 




grade levels. 



Table 3 
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Chi-square Analysis of Observed and Expected Coding Values Using 
Mathematics Materials Analysis Instrument (MMAI) on P.A.S.S. Curricula 



Coding Observed Expected 

Value 1989 1995 1989 1995 



Grades 5-8 



1 


63 


2 


32.5 


32.5 


2 


65 


35 


50.0 


50.0 


3 


40 


77 


58.5 


58.5 


4 


15 


69 


42.0 


42.0 



X 2 = 112.66 a =.05 df=3 



Grades 9-12 
1 


73 


17 


45 


45 


2 


42 


25 


33.5 


33.5 


3 


20 


56 


38 


38 


4 


9 


46 


27.5 


27.5 



X 2 = 81.1 a =.05 df = 3 



The researcher examined the unit titles (Table 4) and tables of content that offered a cursory 
overview of contents in the 1989 and 1995 curricula. This examination further supported the findings 
that the 1995 curriculum was more aligned to the Standards than the 1989 curriculum. 



Table 4 



Unit Titles of P.A.S.S. Curricula 



Unit 


Tide 


Unit 


Tide 


1989 Course 


General Math A 


General Math B 


I 


Numeration Systems and Place Value 


VI 


Fractions 


n 


Addition and Subtraction 


VU 


Decimals 


ffl 


Multiplication 


VIII 


Percent 


IV 


Division 


DC 


Measurement 


V 


Application 


X 


Metrics 





Inteerated Math A 


Integrated Math B 


I 


Number and Number Relationships 


VI 


Statistics and Probability 


n 


Number Systems and Number Theory 


VU 


Algebra 


UI 


Computation and Estimation 


VUI 


Geometry 


IV 


Patterns, Functions, and Math Connections 


DC 


Problem Solving 


V 


Measurement 


X 


Mathematics as 
Communication 
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Descriptive statistics were used to explore the materials in more detail. Central measures of 
tendency were computed from data obtained from MMAI (Appendix A) for the 1989 and 1995 curricula 
(Tables 5-6). The 1989 data in Table 5 clearly show the mean and median measures centering around or 
below 2.0 with small standard deviations (with one exception). This indicates the coders agreed that the 
1989 curriculum represented low levels of content relating to the NCTM Standards . Category H in grades 
5-8 has an exceptionally high standard deviation of 0.77 compared to the other categories. Closer 
examination shows a higher rating by Nud*Ist that clearly affected this standard deviation. This proved to 
be true for most categories in both grade levels. This was not surprising because the computer does not 
forget data and the assignment of ordinal values is dependent upon memory. Human coders are more 
likely to forget specific details and therefore assign lower ordinal values on MMAI. At any rate, the 
group standard deviation remained small showing agreement between the three coders. 

The 1995 data in Table 6 clearly show the mean and median measures centering around or above 
3.0 with small standard deviations (with several exceptions). This indicates the coders agreed that the 
1995 curriculum represented moderate levels of content relating to the NCTM Standards . Categories G 
and H in grades 5-8 have standard deviations of 0.42 and 0.51, and categories A and E have standard 
deviations of 0.44 and 0.63. These measurements are a little higher than the other standard deviations. 
Closer e xaminati on again shows a higher rating by Nud*Ist that clearly affected these standard deviations. 
A gain this seems to indicate the computer coding isolated more applicable text units than the human 
coders were able to observe and remember. 

Figures 1-2 demonstrate the distributions of the group means data in Tables 5-6. The graphs 
confirm the interpretations made from the dispersion measurements. The distributions are fairly 
consistent and the scatterplots depict the higher ratings for the 1995 curriculum in grades 5-8 and 9-12. 

Data collection worksheets provided frequency counts for three major categories: word problems, 
skill and drill problems, and projects and investigations. The category for skill and drill problems was 
chosen as highly representative of problems found in traditional mathematics courses. The category for 
word problems was chosen as representative of problems found in both traditional and integrated courses. 
The category for projects and investigations was chosen as indicative of problems requiring the higher- 
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Table 5 



Measures of Dispersion for Coding Values on MMAI for 1989 P.A.S.S. Curriculum 



MMAI Category 


Coder 1 
Mean 


Coder 2 
Mean 


Coder 3 
Mean 


Group 

Mean 


Standard 

Deviation 


Grades 5-8 












A 


2.08 


2.08 


2.17 


2.11 


0.05 


B 


1.71 


1.86 


1.86 


1.81 


0.09 


C 


2.25 


2.25 


2.25 


2.25 


0.00 


D 


2.44 


2.22 


2.56 


2.41 


0.17 


E 


2.40 


2.60 


2.60 


2.53 


0.12 


F 


1.71 


2.00 


2.29 


2.00 


0.29 


G 


1.00 


1.00 


1.00 


1.00 


0.00 


H 


2.00 


2.00 


3.33 


2.44 


0.77 


Median 


2.08 


2.08 


2.25 


2.11 




Mean 


1.95 


2.00 


2.26 


2.07 


0.17 


Standard Deviation 


0.47 


0.46 


0.67 


0.50 






Coder 1 


Coder 2 


Coder 3 














Group 


Standard 


MMAI Category 


Mean 


Mean 


Mean 


Mean 


Deviation 


Grades 9-12 












A 


1.83 


2.17 


2.33 


2.11 


0.26 


B 


1.67 


2.00 


1.83 


1.83 


0.17 


C 


2.00 


2.00 


1.5 


1.83 


0.29 


D 


1.83 


1.83 


2.17 


1.94 


0.20 


E 


1.33 


1.17 


1.5 


1.33 


0.17 


F 


1.67 


1.83 


2.17 


1.89 


0.26 


G 


1.00 


1.00 


1.00 


1.00 


0.00 


H 


2.00 


1.67 


2.00 


1.89 


0.19 



Median 


1.67 


1.83 


1.83 


1.83 




Mean 


1.67 


1.71 


1.81 


1.73 


0.08 


Standard Deviation 


0.35 


0.42 


0.45 


0.37 
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Table 6 



Measures of Dispersion for Coding Values on MMAI for 1995 P.A.S.S. Curriculum 



MMAI Category 


Coder 1 
Mean 


Coder 2 
Mean 


Coder 3 
Mean 


Group 

Mean 


Standard 

Deviation 


Grades 5-8 
A 


3.42 


3.25 


3.58 


3.42 


0.17 


B 


3.14 


3.57 


3.29 


3.33 


0.22 


C 


3.75 


3.25 


3.50 


3.50 


0.25 


D 


2.78 


2.44 


2.56 


2.59 


0.17 


E 


3.40 


3.20 


3.30 


3.30 


0.10 


F 


3.43 


3.14 


3.57 


3.38 


0.22 


G 


2.22 


3.00 


2.89 


2.70 


0.42 


H 


3.33 


3.00 


4.00 


3.44 


0.51 


Median 


3.37 


3.17 


3.40 


3.36 




Mean 


3.15 


3.09 


3.30 


3.18 


0.11 


Standard Deviation 


0.51 


0.34 


0.47 


0.37 







Coder 1 


Coder 2 


Coder 3 


Group 


Standard 


MMAI Category 


Mean 


Mean 


Mean 


Mean 


Deviation 


Grades 9-12 












A 


3.17 


3.00 


3.83 


3.33 


0.44 


B 


3.00 


3.00 


3.00 


3.00 


0.00 


C 


2.33 


2.50 


2.83 


2.55 


0.25 


D 


2.83 


3.00 


3.00 


2.94 


0.10 


E 


3.50 


2.50 


3.67 


3.22 


0.63 


F 


2.33 


2.67 


2.67 


2.56 


0.20 


G 


2.83 


2.83 


3.00 


2.89 


0.10 


H 


2.83 


2.67 


2.83 


2.78 


0.09 


Median 


2.83 


2.75 


3.00 


2.92 




Mean 


2.81 


2.74 


3.00 


2.85 


0.14 


Standard Deviation 


0.40 


0.21 


0.32 


0.24 
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Figure 1: Distributions of coded values for grades 5-8 of 
P.A.S.S. curricula 
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order-thinking processes envisioned in the Standards, Projects and investigations were combined into 
one category because they had similar requirements relating to time frames and critical thinking 
processes. Percents were computed and interpreted as showing the 1989 curriculum as largely traditional, 
and the 1995 curriculum as more adequately reflecting recommendations in the Standards (Table 7, 
Figures 3-4). 

Word count lists were also analyzed for content categories relating to culture and gender. The 
findings support the vision of the Standards relating to these issues. The 1995 materials were measurably 
superior to the 1989 materials with respect to attention to Latino culture and male/female gender issues. 
The researcher concluded from all of these analyses that the 1995 P. A.S.S. materials measurably and 
qualitatively improve upon the 1989 P.A.S.S. materials in reflecting reform ideas expressed in the 
Standards. 



Study Question 2: The second study question "Can a researcher-designed evaluative instrument 
measure the extent to which curricular materials meet the Standards ?" was answered in terms of content 
validity and interrater reliability. The evaluative instrument Mathematics Materials Analysis Instrument 
(MMAI) (Appendix A) and supplementary attachments (Appendix B-D) were validated for content by a 

panel of three expert educators who are familiar with and experienced in the vision of the NCTM 

< 

Standards . 

The instrument was designed in the fall of 1995 and given to the content validation panel in 
January 1996. It was subsequently edited several times, and final panel validation was received in March 
1996. The establishment of interrater reliability between human coders and computer coding in using the 
instrument on the study materials has already been discussed. In addition, a pilot study conducted with 
two different texts (Cummins, Kenney, & Kanold,1988; Rubenstein, Craine, & Butts, 1995) and five 
independent coders confirmed this reliability (Table 8). The tests found no significant differences at .05 
level of significance between coders. This was true for two independent trials. The researcher concluded 
that coding was performed accurately and reliably by independent coders with the validated MMAI. 
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Table 7 



Frequency Counts and Percentages of Types of Problems for P.A.S.S. Curricula 



Year 


Word 

Problems 


Skill and Drill 
Problems 


Projects and 
Investigations 


1989 


609 


3866 




1995 


430 


1124 


195 


As a percent of total problems 
1989 


14% 


86% 


. 


1995 


25% 


64% 


11% 




Word FYoblems Skill and DrU) Rejects & 

Investigations 



Figure 3: Comparisons of types of problems in P.A.S.S. 
curricula. 
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Figure 4: Comparisons of combined types of problems from 
Figure 3 in P.A.S.S. curricula 



Table 8 
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Kruskal-Wallis H-test to Establish Interrater Reliability Between Coders in Pilot Study 
Using the Mathematics Materials Analysis Instrument (MMAI) 



MMAI Category 


Coder 1 


Text A: Integrated Math 
Rank Coder 2 


Rank 


Coder 3 


Rank 


A 


3.50 


13.5 


3.50 


13.5 


3.50 


13.5 


B 


3.17 


6.5 


3.83 


20.5 


4.00 


24.0 


C 


2.67 


2.0 


3.17 


6.5 


3.50 


13.5 


D 


2.83 


3.0 


3.83 


20.5 


3.33 


9.5 


E 


3.17 


6.5 


3.50 


13.5 


3.67 


17.0 


F 


3.50 


13.5 


3.17 


6.5 


3.83 


20.5 


G 


3.83 


20.5 


3.83 


20.5 


3.83 


20.5 


H 


2.50 


1.0 


3.00 


4.0 


3.33 


9.5 




N, = 8 




00 

II 

(N 




N 3 = 8 








R.um= 66.5 




R*um = 105.5 




R«um = 128 


H = 4.841 a =.05 






Text B: Informal Geometry 






MMAI Category 


Coder 1 


Rank 


Coder 2 


Rank 


Coder 3 


Rank 


A 


1.33 


6.0 


1.67 


14.5 


1.83 


20.0 


B 


2.00 


23.0 


1.83 


20.0 


2.17 


24.0 


C 


1.17 


2.5 


1.83 


20.0 


1.50 


9.5 


D 


1.67 


14.5 


1.33 


6.0 


1.67 


14.5 


E 


1.33 


6.0 


1.17 


2.5 


1.17 


2.5 


F 


1.50 


9.5 


1.17 


2.5 


1.67 


14.5 


G 


1.67 


14.5 


1.83 


20.0 


1.67 


14.5 


H 


1.83 


20.0 


1.50 


9.5 


1.50 


9.5 




N, = 8 




N 2 =8 




N 3 = 8 








R*um= 96 




R.um=95 




Reum = 109 



H = 0.305 a =.05 
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Conclusions 

This study found the 1995 Portable Assisted Study Sequence (P.A.S.S.) curricular materials to be 
measurably superior to the 1989 curricular materials with respect to meeting Standards ideals and 
recommendations. It also affirmed that an evaluative instrument could be designed to effectively measure 
the extent to which mathematics curricular materials meet the Standards . The Mathematics Materials 
Analysis Instrument (MMAO (Appendix A) designed for this study was proven to be reliable and able to 
effectively measure the P.A.S.S. curricula. 

Significance of Findings 

This study can benefit educators and society on many levels. The P.A.S.S. program is 
continuously updated, and this important first course of the mathematics sequence can provide direction to 
further curriculum developers. This can result in a new curriculum for migrant students designed around 
the concepts and transitions inherent in the NCTM Standards . This can have great impact nationally due 
to the role of the California P. A.S.S. office in providing direction and curriculum for the nation. 

The evaluative instrument designed for this study (Appendix A) can be used as is, or can be 
revised and improved with further research. It can provide guidance and direction during the process of 
curriculum development as well as for curriculum selection. This can benefit teachers on a local, state, 
and national level. 

Content analysis has not been used extensively in the mathematics education field, and this study 
can provide insights and direction for researchers wishing to utilize this methodology in mathematics 
education. The process for designing and validating the evaluative instrument can also have value to 
future researchers. This study succeeded in quantifying the subjective recommendations in the NCTM 
Standards and this is also of value to educators and researchers on many levels. 

This study is a beginning-the 1989 Standards are not the final document in mathematics reform. 
California is one of many states rigorously pursuing state standards in all core disciplines. The analysis of 
new curricular materials is essential in order to produce materials that meet newly recommended 
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standards. This study demonstrates one way to measure the success of any reform efforts and can be 
beneficial across many disciplines. Significant curricular reform will have a significant social impact. 

This study represents a pioneering effort to quantify changes in our mathematics curriculum 
through the design and introduction of the evaluative instrument, Mathematics Materials Analysis 
Instrument (MMAI) (Appendix A). Future research can improve the validity and reliability of this 
instrument through rigorous statistical treatments such as factor analysis, and more complex studies to 
improve interrater reliability and content validation. Interrater reliability studies using participants from 
dissimilar environments holding varied philosophical viewpoints could improve the reliability of MMAI. 
The instrument can be further streamlined, for example, by consolidation of categories. The study can be 
replicated with other curricular materials and in other disciplines to further strengthen its effectiveness 
and usefulness to education. The research design for this study can be used to analyze other educational 
issues. Finally, the power of fourth-generation content analysis software such as Nud*Ist has not been 
fully utilized in this study. Fourth-generation programs offer opportunities for qualitative theory building. 
Future researchers can unleash this power and build immensely on this pioneering study. 
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A. Problem Solving (Critical-Thinking Skills) 
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D. Measurement 

Mathematics should be presented as having power , usefulness , and creative aspects so it is not viewed by students as a static, bounded set of rules and 
procedures to be memorized but quickly forgotten. When measurement is explored through rich, investigative, purposeful activity, it affords such opportunity. 
Fundamental concepts of measurement should be demonstrated through concrete experiences. 
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F. Geometry 

Students should have knowledge of concepts such as parallelism, perpendicularity , congruence, similarity, and symmetry. They should know properties of 
simple plane and solid geometric figures and should be able to visualize and verbalize how objects move in the world around them using terms such as slides ; flips ; 
and turns. Geometric concepts should be explored in settings that involve problem solving and measurement 
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G. Probability and Statistics 

Understanding probability and the related area of statistics is essential to being an informed citizen and is important in the study of many other disciplines. 
Students in grades 5-8 have a keen interest in trends in music , movies, and fashion and in the notions of fairness and the chances of winning games. These interests 
can be excellent student motivators for the study of probability and statistics. 
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H. Algebra 

One of the most important roles of the middle grade mathematics curriculum is to provide a transition from arithmetic to algebra. It is crucial that students 
in grades 5-8 explore algebraic concepts in an informal way in order to build a foundation for the subsequent formal study of algebra. 




t/i 

<D 

I 

T3 

<D 

-o 

O 

U 

Cm 

O 

S 

G 

C/3 



Cm 

O 

S 

3 

a> 



bO 

a 
• ^ 

T3 

G 

tc 

G 

a> 

X 

-M 

*G 

cd 



O 

o 

X! 

§ 

<2 

C/3 

a 



60 

C 



3 

"3 

o 

X 

C/3 

a> 

13 

> 

T3 

a> 

T3 

O 

a s 

O o 

S 5 

G cd 



C/3 



3 



0> 

X cd 

■M O 

<d a> 
ti 00 

2 o 

S ^ 

o 

O 



00 






05 




<D TD 



* 

:s 

§ 

a 

<D 

3 

03 



o 

in 



Jl 

r"* /-■* 



■C on 



M l _3 »■* 

« St# 

•SS " 

*s 

Cd 
CA 

13 

•c 

4> 

« 

s 

03 

U 

ce 

s 

a> 

5 

« 

S 



N 

I 

as 



»’E 



<N 



ON 



J3 M 



cti 



& 



o ,a> 

« ta 

s jg 






g j§ .2 £ 

’§>' 'I l-s 

« id « § 



C/3 

i 

> 

i 

8 

u 




top 

ai 

03 

13 

I 

I 



a 

<L> 

a 

3< 

§* 

co 



o 

Id 

03 

+-► 

o 

<u 

’5* 

a- 

t\ 

o 

0 

1 

£ 



t-H 

a> 

03 

jj g 

g £ 



JD 


#52 


#52 




*s 


>4— > 


H 


H 


H 


r—H 


CN 


CO 



§ 

.a 

3 

£ 

o 

<L> 

c3 

Q 






03 

1-4 

a> 

a> &• 
toO S 
cd ^ 
CLh u 



o 

£ 



o 

fc 



O 

ERIC 



Problem solving (Critical-Thinking Skills) 
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The 9-12 program should give students opportunities to develop, learn , and use the language and notation of mathematics. Vocabulary that is unique to 
mathematics and terms that have a common, as well as a mathematical, connotation should be developed and used throughout the curriculum. Mathematical ideas 
should be expressed by writing, speaking, making models, drawing diagrams, and preparing graphs. Opportunities should be provided for discussing mathematical 
topics. 
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C. Computation and Estimation 

The 9-12 program gives students a variety of opportunities to gain facility in computing with whole numbers, decimals, and fractions and in using the four 
basic operations . It also provides opportunities for students to develop and use estimation skills and concepts on a continuing basis throughout the materials. 
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D. Reasoning 

Provision is made at all levels for introducing and using simple valid arguments. The 9-12 program gives students opportunities to learn the basic tenets of 
logical argument and to validate arguments. Connections among various representations of mathematical ideas are used to develop arguments. 

To what extent does this happen in your materials? None Lo Mod Hi 
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G. Technology 

The 9-12 program should use calculators and computers as tools for graphing, problem solving, performing tedious calculations, generating data, and 
developing concepts. Materials should allow students to appropriately choose calculators or computers to perform calculations that warrant their use. 
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Appendix B 



Instructions for using Mathematics Materials Analysis Instrument (MMAI) 
Materials on hand: 

1) Scope and Summary of National Council of Teachers of Mathematics (NCTM) 

Standards - 6 pages 

2) General Coding Rubric for Mathematics Materials Analysis Instrument (MMAI) 

3) Mathematics Materials Analysis Instrument (MMAI) 

- Grades 5-8 (pp. 1-9) 

- Grades 9-12 (pp. 10-18) 

4) Worksheet (pp. 1-4) with Example-Grades 9-12 (pp. 5-8) 



1) Preview the above materials to understand the topics and areas involved in the NCTM 
Standards and the MMAI. The evaluator can refer to the reference materials listed below for 
further clarification. 

2) Review the mathematics curricular materials and supplementary materials that are being 
evaluated to obtain a vision of the contents, and to provide insight into the scope and direction 
of the content and objectives. 

3) Complete the MMAI. The General Coding Rubric should be used to help focus on general 
considerations that are part of the vision of the NCTM Standards . A coding value from 

1 to 4 is circled on the instrument for each subcategory. The materials listed at the top of this 
page, the mathematics curricular materials, supplementary materials, and the reference 
materials listed below may be referred to as often as necessary during the completion of the 
instrument. 

4) The worksheet is completed after the instrument is completed. The Sum of coded values for 
each category is transferred to the appropriate section on the worksheet. The worksheet pro- 
vides details for calculating and interpreting the results. 



Recommended reference materials: 

Mathematical Sciences Education Board. (1989). Everybody counts. Sacramento, CA: Author. 

National Council of Teachers of Mathematics. (1995). Assessment standards for school 
mathematics Reston, VA: Author. 

National Council of Teachers of Mathematics. (1995). Connecting mathematics across the 
curriculum . Reston, VA: Author. 

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards 
for school mathematics. Reston, VA: Author. 



Appendix C 

General Coding Rubric for Mathematics Materials Analysis Instrument (MMAI) 

To what extent is this represented in your curriculum? 



1-No 


Not represented 


2- to 


Low level of representation 


3-Mod 


Moderate level of representation 


4-Hi 


High level of representation 



The following considerations are important in determining which code is most 
applicable. 

1 - No • Traditional - non-integrated. Mathematics is presented in a linear fashion, i.e.. 

Algebra, Geometry, Trigonometry, and so on. 

• Rote learning, memorization, deductive reasoning is emphasized. 

• Problems are close-ended; computational skills are emphasized. 

• Calculators may be optional. 

• Teacher is the expert and students are encouraged to work alone. 

• The "decreased attention" topics are emphasized. (See "Scope and Summary of 
NCTM Standards - Summary of Changes in Content and Emphases). 

2 - Lo • Traditional - non-integrated. Mathematics is presented in a linear fashion, i.e.. 

Algebra, Geometry, Trigonometry, and so on. 

• Rote learning and memorization are emphasized but there is some flexibility. 

• Problems are more complex and alternate solutions occasionally exist. 

• Teacher is the expert but students are encouraged at times to work together. 

• Computers and calculators are encouraged for computational exercises. 

3- Mod • Integrated mathematics curriculum (broad range of topics within mathematics). 

• Students work periodically in cooperative groups. 

• Projects, portfolios, manipulatives, and models are used to a limited degree. 

• Computers and calculators are used for exploration as well as computational 
exercises. 

• Students use several methods to communicate their ideas. 

• The teacher and students share the "expert" role, but the teacher is the 
ultimate authority. 

4- High • Integrated mathematics curriculum (broad range of topics within mathematics). 

• Interdisciplinary curriculum. 

• Teacher is facilitator, provides resources, and introductory information. 

• Students are team members, explorers, discoverers, and predictors. 

• Computers, calculators (including graphing calculators), and multimedia are 
used extensively. 

• Concrete models and manipulatives are available or are constructed by students to 
explore and refine ideas. 

• Real-world applications are emphasized; students are encouraged to explore in their 
own community. 

• Projects and investigations replace rote exercises. 

• Students learn to compute through rote exercises but quickly advance to more 
complex ideas and problems. 

• Problems emphasize open-ended responses. 

• Students use a variety of communication methods. 
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Appendix D 



WORKSHEET for Coding MMAI 

Title: 

Publisher: 

Date of Pub.: 



Enter the Total Sums of Coded Values from each subcategory on the MMAI to the appropriate 
curriculum section Grades 5*8 or 9-12. 



No. of Sum of 

Tt.pms Curriculum: 5*8 Coded 

Values 



12 

7 

4 

9 

10 

7 

9 

_3 

61 



A. Problem Solving (Critical-Thinking Skills) 

B. Communication 

C. Computation 

D. Measurement 

E. Number and Number Systems 

F. Geometry 

G. Probability and Statistics 

H. Algebra 



Total: 



No. of Sum of 

Tt.pma Curriculum: 9-12 Coded 

Values 



6 A. Problem Solving (Critical-Thinking Skills) 

6 B. Communication 

6 C. Computation and Estimation 

6 D. Reasoning 

6 E. Integration 

6 F. Interdisciplinary Emphasis 

6 G. Technology 

6 H. Other Curriculum Emphasis 

48 




83 



i 



Total: 



CALCULATIONS and INTERPRETATIONS 



This instrument can be used by one evaluator or by a team of two or more evaluators. The 
calculations in subheading I apply to both cases. The calculations in subheading II are to be used 
for two or more evaluators and should be made in addition to those made in subheading I. 

I: One Evaluator: 

Finding the mean ( x ) for all categories : 

T = Total Sum of Coded Values 

A = No. of Applicable Items: 

[ Grades 5-8: (A = 61) or Grades 9-12: (A = 48) ] 

Enter the sums from the worksheet: 

T = A = 

T 

X — — = (to at least 3 decimal places) 

A 

Interpretation: The mean indicates the degree of movement toward the Standards . Compare it 
to the Coding Values: 1= None; 2= Low; 3 = Moderate; 4 = High. 

Conclusions: 



Finding the mean for each subcategorv : Further statistical tests can be done on each sub- 
category. The easiest comparisons can be made by simply finding the median and mean of each 
subcategory and comparing them to the Coding Values: 1 = None; 2 = Low; 3 = Moderate; 

4 = High. This is a quick check to find weaknesses and strengths within the categories. 

(For Grades 9-12, use the formula shown below. For Grades 5-8, refer to page 1 of the worksheet 
and change the 6 in the formula to match the number of items in each subcategory.) 






Conclusions: 



Formula Mean 

A = 

B = 

C = 

D = 

E = 

F = 

G = 

H = 



Interpretation: 

Categories with means above 3.0 can be seen to be moving toward the Standards and can be 
compared to the Coding Values: 3 - Moderate and 4 - High . Categories below 3.0 can be 
compared to the Coding Values: 1 - None f 2 - Low and 3 - Moderate . The materials will need to 
be supplemented in these categories with activities and exercises reflecting higher movement 
toward the Standards . Coding Values for each item can be examined in these categories to help 
in determining the type of supplementary activities that will be needed. 
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II: Two or More Evaluators: 



Finding the mean ( X ) and Standard Deviation (SD or Cf) for all categories: 
n = number of evaluators 

i = 1, 2, 3, ... n 

Ti = Total Sum of Coded Values for Evaluator i 

Ai = No. of Applicable Items for Evaluator i: 

[ Grades 5-8: (Ai = 61) or Grades 9-12: (Ai = 48) ] 



i Ti 


Ai 


1 




2 




3 




4 




n 




Total 




Ts 


As 


Ts= 2 Ti 


n 

As = y. Aj 

i 


T s = 


As = 


— Ts 

X (group mean) : — = 


(to at least 3 decimal places) 


xi = x (mean) from calculations in Part I for each evaluator 






v n-1 




a = 





Interpretation: 

The mean indicates the degree of movement toward the Standards . Compare it to the Coding 
Values: 1= None; 2= Low; 3 = Moderate; 4 = High. 

Conclusions: 




3 



II. Continued 



The standard deviation ( SD or a ) measures the distribution of data in relationship to the mean. 
A small SD indicates the total sum of coded values are close together which means the evaluators 
are in close agreement in their opinions of the materials. A large SD indicates that the total sum 
of coded values are spread out which indicates the evaluators are not in close agreement in their 
opinions of the materials. 



The mean and SD also indicate the percentage of values in a normal distribution: 

X ± 1.0 SD = approximately 68% of the total coded values 

X ± 2.0 SD = approximately 95% of the total coded values 

X ± 2.5 SD = approximately 99% of the total coded values 

X ± 3.0 SD = approximately 99 + % of the total coded values 



Conclusions: 



EXAMPLE - Grades 9-12 

WORKSHEET 

Title: INTEGRATED MATH 
Publisher: XYZ PUBLICATIONS 
Date of Pub.: 1995 



Enter the Total Sums of Coded Values from each subcategory on the MMAI to the appropriate 
curriculum section Grades 5-8 or 9-12. 



No. of Sum of 

Items Curriculum: 5-8 Coded 

Values 

12 A. Problem Solving (Critical-Thinking Skills) 

7 B. Communication 

4 C. Computation 

9 D. Measurement 

10 E. Number and Number Systems 

7 F. Geometry 

9 G. Probability and Statistics 

_3 H. Algebra 

61 

Total: 



No. of 

T tarns Curriculum: 9-12 



Sum of 
Coded 
Values 



6 

6 

6 

6 

6 

6 

6 

6 

48 



A. Problem Solving (Critical-Thinking Skills) 

B. Communication 

C. Computation and Estimation 

D. Reasoning 

E. Integration 

F. Interdisciplinary Emphasis 

G. Technology 

H. Other Curriculum Emphasis 



21 

16 

15 

11 

18 

11 

22 

15 

Total: 141 




5?3 



EXAMPLE - Grades 9-12 
CALCULATIONS and INTERPRETATIONS 
This instrument can be used by one evaluator or by a team of two or more evaluators. The 
calculations in subheading I apply to both cases. The calculations in subheading II are to be used 
for two or more evaluators and should be made in addition to those made in subheading I. 

I: One Evaluator: 

Finding the mean x for all categories: 

T = Total Sum of Coded Values 

A = No. of Applicable Items: 

[ Grades 5-8: (A = 61) or Grades 9-12: (A = 48) ] 

Enter the sums from the worksheet: 

T =141 A = 48 



X = — = 2.938 (to at least 3 decimal places) 

A 

Interpretation: The mean indicates the degree of movement toward the Standards . Compare it 
to the Coding Values: 1= None; 2= Low; 3 = Moderate; 4 = High. 



Conclusions: This material is moving toward the Standards as indicated by the mean of 2.938 compared 
to 2.0 - Low and 3.0 - Moderate. 



Finding the mean for each subcategory : Further statistical tests can be done on each 
subcategory. The easiest comparisons can be made by simply finding the mean of each 
subcategory and comparing it to the Coding Values: 1 = None; 2 = Low; 3 = Moderate; 
4 = High. This is a quick check to find weaknesses and strengths within the categories. 








6 . 

Formula 




Mean 


A 


21/6 


= 


3.50 


B 


16/6 


= 


2.67 


C 


15/6 


= 


2.50 


D 


17/6 


= 


2.83 


E 


18/6 


= 


3.00 


F 


17/6 


= 


2.83 


G 


22/6 


= 


3.67 


H 


15/6 


= 


2.50 



Conclusions: 



Categories A. B. E. F. and G are moving 


toward the Standards and can be 


compared to the coding values 3 - 


Moderate and 4 - Hieh. Categories B. G 


D. F. and H can be compared to the 


Coding Values: 2 - Low and 3 - 


Moderate. The materials will need to be 


supplemented with activities and 


exercises reflecting higher movement 


toward the Standards in these categories. 


Further examination of the items in each 


category will indicate the specific areas 


to be targeted. 





Interpretation: 

Categories with means above 3.0 can be seen to be moving toward the Standards and can be 
compared to the Coding Values: 3 - Moderate and 4 - High . Categories below 3.0 can be 
compared to the Coding Values: 1 - None , 2 - Low , and 3 - Moderate . The materials will need to 
be supplemented in these categories with activities and exercises reflecting higher movement 
toward the Standards . Coding Values for each item can be examined in these categories to help 
in determining the type of supplementary activities that will be needed. 



II: Two or More Evaluators: 



EXAMPLE - Grades 9-12 



Finding the mean ( X ) and Standard Deviation (SD or a) of all categories: 
n = number of evaluators 
i = 1, 2, 3, ... n 

Ti = Total Sum of Coded Values for Evaluator i 
Ai = No. of Applicable Items for Evaluator i: 

[ Grades 5-8: (Ai « 61) or Grades 9-12: (A} = 48) ] 

(For purposes of this example , assume three additional worksheets for Calculation I ( Grades 9-12) have 
been completed for three additional evaluators . The totals for Ti are entered from the four worksheets). 



i 


Ti 


Ai 


i 


141 


48 


2 


166 


48 


3 


126 


48 


4 


175 


48 



Total 


608 


192 




T s 


As 



Ts= Z T i 

i 



As c Aj 

i 



T s = 608 Ag = 192 



X (mean) : — = 3.167 (to at least 3 decimal places) 

As 

xi = X (mean) from calculations in part I for each evaluator 



For purposes of this example , assume the calculated means x, are as follows: 

Xi =3.065 x ? =3.458 x 3 - 2.930 x* - 3.723 




Interpretation: 

The mean indicates the degree of movement toward the Standards . Compare it to the Coding 
Values: 1= None; 2= Low; 3 = Moderate; 4 = High. 

Conclusions: This material is moving toward the Standards as indicated by the mean of 3. 167 as 
compared to 3.0 - Moderate and 4.0 - High. 



EXAMPLE - Grades 9-12 

II: Two or More Evaluators (Continued): 

The standard deviation ( SD or O ) measures the distribution of data in relationship to the mean. 
A small SD indicates the total sum of coded values are close together which means the evaluators 
are in close agreement in their opinions of the materials. A large SD indicates that the total sum 
of coded values are spread out which indicates the evaluators are not in close agreement in their 
opinions of the materials. 

The mean and SD also indicate the percentage of values in a normal distribution: 

X ± 1.0 SD — approximately 68% of the total coded values 

X ± 2.0 SD = approximately 95% of the total coded values 

X ± 2.5 SD = approximately 99% of the total coded values 

X ± 3.0 SD = approximately 99 + % of the total coded values 

Conclusions: The mean 3. 167 ± 0.39 = 3.557 or 2. 778. This means 68% of the total coded values are 
between 2. 778 and 3.557. These values can be compared to the Coding Value: 3 - Moderate. Further- 
more. the mean 3. 167 ± 2.0 SD = 3. 167 ± 2.0 x 0.39 = 3.947 or 2.387. This means 95% of the total 
coded values are between 2.387 and 3.947. The materials therefore seem to be movinz toward the 
Standards. 
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